27/04/23 - IVOA Datalink

Notes on using DaCHS for this:

May need to use other method (connecting to external database & importing rather than just a view on table): https://dachs-doc.readthedocs.io/refdoc.html#harvesting-from-remote-databases

Products table: https://dachs-doc.readthedocs.io/refdoc.html#using-fromdbupdating-on-ignoresources

embeddedGrammar (parse an api w/ replica locations?) https://dachs-doc.readthedocs.io/ref.html#element-embeddedgrammar

maybe something like this:

need some way of passing replica locations into external database, or expose an api
this then needs to be populated in a products table (and grouped with some id somehow?)
datalink needs to expose this

Reasons not to use trigger:

maint overhead in running it alongside rucio
db overhead in running it for every row (not part of rucio core infra, so they're not responsible)
can't populate across database, so would need a poller service working through the resulting table
difficulty in getting replica paths?

Looks like a bespoke datalink service would be better:

https://wiki.ivoa.net/internal/IVOA/InterOpApr2022CSP/ESO_experience_adopting_VO_CSP_IVOA_April2022.pdf

Could be "redirect mode" or "datalink" (actually datalink only supports one #this anyway, so would need to only use one in datalink resp). Redirect wouldn't work with tokens.

Needs to talk to site directory, use client ip and geoip to get closeness.

Checked out SODA ImgServ for LSST (python) on github -> bespoke SODA implementation. Looks easy enough.

26/04/23 - IVOA Datalink

Looking at using products table in DaCHS:

root@b45f6cd5518b:/var/gavo/inputs/rucio# cat products.rd 
<resource schema="rucio">
  <table id="withdate" mixin="//products#table" onDisk="True" primary="accref" forceUnique="True" dupePolicy="overwrite">
    <column name="mtime" type="timestamp" ucd="time;meta.file" tablehead="Timestamp" description="Modification date of the source file."/>
  </table>

  <data id="import" updating="True">
  <sources pattern="data/*.fits">
    <ignoreSources
      fromdbUpdating="select accref, mtime from \schema.withdate"/>
  </sources>
  <fitsProdGrammar>
    <rowfilter procDef="//products#define">
      <bind key="table">"\schema.withdate"</bind>
    </rowfilter>
  </fitsProdGrammar>
  <make table="withdate">
    <rowmaker>
      <map key="mtime">datetime.datetime.utcfromtimestamp(
        os.path.getmtime(\fullPath))</map>
    </rowmaker>
  </make>
  </data>
</resource>

25/04/23 - IVOA Datalink

24/04/23 - More RSE endpoints and elastic service migration

Added CHSRC_XRD. Some issues with xrd checksumming.

For elastic migration, tried to use reindex endpoint but needed to whitelist remote. Cinder volumes now stuck in deleting and new ones won't provision due to STFC openstack Cinder issue.

Moved rest of deployment yamls across to management in /opt/src-services-deployment.

21/04/23 - AL

20/04/23 - Migrating elastic service

Spent most of the day trying to rollback a helm update of the bitnami/elasticsearch chart (as old version had been deprecated and needed to set settings in helm chart to enable snapshots).

19/04/23 - Migrating grafana service

aliased rucio.srcdev.skao.int, monit.srcdev.skao.int domains pointing towards haproxy instance, renewed ssl cert to include these subdomains so they're ready for migration of services
altered capi_workload_deploy playbook to exclude the kubeconfig role (the one that checks if the workload cluster is up and running and gets the workload kubeconfig), instead opting to create a new "custom" role with the same name that doesn't use breaking relative paths. copied most of the bits from the services role that worked, but also creates a script on the management machine that creates aliases (k9s/helm/kubectl) for the management/workload cluster
created capi_workload_post playbook that uses the services k8s.ingress role to deploy ingress. tested & added to makefile target
tested cinder, works.
moved everything to repo, updated README
deployed monit.srcdev.skao.int/grafana (added ingress and dashboards to values yaml)
deployed monit.srcdev.skao.int/elastic (behind basic auth)

18/04/23 - Ansiblifying CAPI

Continued..

17/04/23 - Ansiblifying CAPI

Continued..

14/04/23 - AL

13/04/23 - Ansiblifying CAPI

Continued..

12/04/23 - Ansiblifying CAPI

Continued..

11/04/23 - Ansiblifying CAPI

Continued..

10/04/23 - Ansiblifying CAPI

Continued..

07/04/23 - AL

06/04/23 - Ansiblifying CAPI

Continued..

05/04/23 - Ansiblifying CAPI

Continued..

04/04/23 - Ansiblifying CAPI

Continued..

03/04/23 - Ansiblifying CAPI

Continued..

Top